Conventional computerized devices such as personal computer systems, workstations, minicomputers and mainframes require the ability to transmit data between components within, and attached to such computerized devices at very high rates of speed. As an example, consider a typical conventional personal computer system containing a processor, a memory system and a variety of peripheral input output components such as storage devices (e.g., floppy disks, hard disks, CD-ROM drives, etc.), network interface connections (e.g., modems or Ethernet network interface cards), video display devices, audio output devices (e.g., soundcards) and so forth. A data bus that interconnects such components within the computer system allows such components to exchange data with each other and allows one component, such as a processor, to control operation of another component such as a memory system or a video display card. Generally, a data bus or interconnect architecture includes a collection of communications hardware and physical data links or connections that couple various devices or components within a computer system environment, as well as software or firmware which operates one or more input output protocols to control communications over the communications hardware and data links. One type of data bus that computer and devices designers commonly use to interconnect components with computer systems and computerized devices is called as a Peripheral Component Interconnect (PCI) bus.
A PCI bus provides a shared bus architecture that allows a processor such as a central processing unit (CPU) operating within the computer system to control access to the PCI bus by components that need to transmit data on the bus. The PCI bus architecture operates at a preset or predefined speed (e.g., 100-Mhz) and forces components on the PCI bus to share the total available bus bandwidth using various bus arbitration algorithms. While this approach is acceptable in many computing system environments, use of a PCI bus to exchange data between components in the computer system can encounter signal integrity and timing constraints that can limit the total speed available for communications between computerized device components. In addition, a PCI bus is fixed in physical size (e.g., 32 bits, 64 bits, 128 bits) and does not scale well to allow for the addition of significantly large amounts of other devices onto the bus beyond a number of bus interface connections or “slots” that a designer initially provides in the computer system. Due to such limitations and increasing requirements of modern day computer applications, computer engineers have developed another type of expandable data bus or interconnect architecture called Infiniband.
Infiniband is an industry standard, channel-based, switched fabric, interconnect architecture for computer systems such as servers and peripherals devices such as storage devices, network interconnects, memory systems, and the like. The conventional Infiniband architecture operates much like a computer network in that each component, peripheral or device that operates in (i.e., that communicates over) an Infiniband architecture or network is equipped with an Infiniband channel adapter that provides input output onto one or more Infiniband communications channels or links (i.e., physical links) that can be coupled to Infiniband switches or that can directly couple to other Infiniband adapters. Infiniband supports a 2.5 Gbps wire-speed connection in each direction on each wire and allows three different performance levels (1×, 4× and 12×) that correspond to three different possible physical connectivity characteristics between the channel adapters. For the 1× performance level which is the lowest performance available in Infiniband, there is one physical data link, wire or connection between adapters (for the total single wire bandwidth of 2.5 Gbps in each direction), whereas the 4× performance level provides four physical parallel links between adapters (for a total bandwidth of 12 Gbps in each direction), and the 12× performance level provides twelve physical parallel links between adapters (for a total bandwidth of 30 Gbps in each direction).
When a component equipped with an Infiniband adapter begins operation, the Infiniband input output protocol operating in the adapter (i.e., operating according to the Infiniband standard) performs an initialization sequence to detect how many Infiniband channels (i.e., physical links) are connected between the adapter and a remote device such as an Infiniband switch to which that adapter is coupled. The Infiniband initialization sequence can query each Infiniband port in the channel adapter and can store the detected number of physical links for each port in a respective configuration parameter for each port called the “LinkWidthEnabled” configuration parameter. The initialization process stores this physical connectivity characteristic for future use during operation of the Infiniband protocol. The LinkWidthEnabled configuration parameter thus stores and defines or describes the actual physical connectivity characteristic of that channel adapter to a remote location, for each port on the channel adapter. In other words, the LinkWidthEnabled configuration parameter for a port identifies the number of physical Infiniband channels or links (e.g., 1×, 4× or 12×) that exist, for example, between a port in a target channel adapter and another port in a host channel adapter or a port in an Infiniband switch.
Thereafter, when data communications between a device equipped with the Infiniband channel adapter is to take place, an Infiniband initialization handshaking sequence occurs between the Infiniband channel adapters to establish, among other things, the proper bandwidth to use for communications. As an example, a host channel adapter can query a target channel adapter for the value of its LinkWidthEnabled configuration parameter to determine what bandwidth or data communications performance level (i.e., 1×, 4× or 12×) that target channel adapter supports. Recall from above that the target channel adapter sets its LinkWidthEnabled parameter during a startup configuration or initialization sequence of that channel adapter.
By way of example, suppose an Infiniband host channel adapter is coupled to an Infiniband switch with twelve Infiniband links or channels (i.e., is coupled in a 12× Infiniband configuration) to provide a server that uses that host channel adapter a maximum performance level on the Infiniband fabric. A storage device including an Infiniband target channel adapter may be coupled to the same Infiniband switch with only four Infiniband links or channels (i.e., in a 4× configuration). When the host channel adapter in the server wants to communicate with the target channel adapter in the storage device, the Infiniband input output protocol operating in the Infiniband host channel adapter performs the initial handshaking operation and queries the Infiniband target channel adapter in the storage device to determine the amount of Infiniband links with which that storage device (i.e., via its target channel adapter) is coupled to the Infiniband switch. The Infiniband input output protocol operating in the target channel adapter in the storage device consults its LinkWidthEnabled configuration parameter (automatically configured during the initialization setup of the adapter in the storage device, as explained above) to reply to the host channel adapter with how many physical Infiniband links (4× in this example) exist between the target channel adapter (in the storage device) and the Infiniband switch. In this manner, upon receipt of the reply containing the LinkWidthEnabled configuration parameter value (e.g., 4×) from the target channel adapter, the host channel adapter can determine that the target channel adapter is only configured in a 4× configuration. As a result, the Infiniband host channel adapter knows it can only communicate with the target channel adapter at a 4× performance level and will “step down” its data communications performance to the target channel adapter from the 12× performance level to the 4× performance level. During Infiniband data communications, the host channel adapter will only use four of the available twelve Infiniband channels or links to communicate with the target channel adapter.